[SPARK-55790][Geo][SQL] Build a complete SRS registry using PROJ 9.7.1 data#54571
[SPARK-55790][Geo][SQL] Build a complete SRS registry using PROJ 9.7.1 data#54571uros-db wants to merge 5 commits intoapache:masterfrom
Conversation
uros-db
left a comment
uros-db
left a comment
There was a problem hiding this comment.
@szehon-ho @cloud-fan Please review.
uros-db
left a comment
uros-db
left a comment
There was a problem hiding this comment.
Docker integration failure seems unrelated to these changes.
szehon-ho
left a comment
szehon-ho
left a comment
There was a problem hiding this comment.
- the pr has two generatedcsv which is very big. just wondering, is there no way to have one generated one, and copy it at build time?
- do we plan to run it just manually? I guess proj does not release that frequently that its worth to automate it?
There probably is a way, although I kind of like keeping these CSV files separate for JVM and Python. They are different modules, and geospatial types themselves live separately, so I think it would make sense to generate both in one go.
Yes, I think that we should be able to run this script manually and inspect the diff (e.g. new SRIDs) per version upgrade, and yes - PROJ releases aren't very frequent. We would otherwise need either CMake or SLQLite to fully automate this. |
szehon-ho
left a comment
szehon-ho
left a comment
There was a problem hiding this comment.
Two small suggestions below.
szehon-ho
left a comment
szehon-ho
left a comment
There was a problem hiding this comment.
lgtm, just some remaining AI assisted review comment for consideration
szehon-ho
left a comment
szehon-ho
left a comment
There was a problem hiding this comment.
thanks, sorry i missed some place below i wanted to get clarification for
|
This looks good to me, @cloud-fan can you also take a look? |
2 participants
What changes were proposed in this pull request?
Build a more complete Spatial Reference System (SRS) registry in Spark, introducing 10000+ entries sourced from the PROJ library's EPSG and ESRI databases, which will unlock substantially improving the breadth of geospatial type support.
Why are the changes needed?
This will enable Geometry and Geography types to support 10000+ additional SRID/CRS values, both in JVM and Python. Note that currently Geometry and Geography types offer only limited SRID support (a few hardcoded values).
Does this PR introduce any user-facing change?
No.
How was this patch tested?
The script and auto-generated golden files are self-contained.
Was this patch authored or co-authored using generative AI tooling?
Yes, Claude 4.6 Opus.